Mott criticality and pseudogap in Bose-Fermi mixtures 
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We study the Mott transition of a mixed Bose-Fermi system of ultracold atoms in an optical lattice, 
where the number of (spinless) fermions and bosons adds up to one atom per lattice, Uf + ub = 1. 
For weak interactions, a Fermi surface coexists with a Bose-Einstein condensate while for strong 
interaction the system is incompressible but still characterized by a Fermi surface of composite 
fermions. At the critical point, the spectral function of the fermions, A(k, oS), exhibits a pseudo- 
gapped behavior, rising as \uj\ at the Fermi momentum, while in the Mott phase it is fully gapped. 
Taking into account the interaction between the critical modes leads at very low temperatures either 
to p-wave pairing or the transition is driven weakly first order. The same mechanism should also 
be important in antiferromagnetic metals with a small Fermi surface. 



A recent experiment with an ultra-cold mixture of 
bosonic and fermionic Yb atoms in an optical lattice 1 
has found a remarkable quantum phase that can be de- 
scribed as a mixed Mott insulator. Such a state 2-4 is es- 
tablished in the strongly interacting regime when the av- 
erage site occupation of the bosons and fermions together 
is an integer, ns+np = 0, 1, .... While the state is incom- 
pressible and hence fluctuations of the total density are 
gapped, the fermions can still move around by exchang- 
ing with the spinless bosons. Hence the mobile objects 
are bound states of a fermionic atom and a bosonic hole. 
Depending on their effective interactions, these bound 
states can form a number of different phases, including 
a Fermi liquid or a paired condensate. But those are 
rather strange fluids, made of composite fermions that 
carry zero net particle number. Accordingly, the spectral 
function of the original fermionic atoms will not display 
a quasi-particle peak. This phase, established for suffi- 
ciently strong interactions, should be contrasted with the 
weakly interacting limit where the fermionic atoms form 
a conventional Fermi sea coexisting with a Bose conden- 
sate(BEC) of the other species. In this paper, we investi- 
gate the quantum phase transition from the incompress- 
ibe mixed Mott state to the compressible metal/BEC 
phase and the fate of the Fermi surface across the tran- 
sition. 

In most solid state systems the Mott quantum criti- 
cal point from a metal to an insulating state is masked 
by antiferromagnetism. In cases where frustration sup- 
presses magnetism, however, it has been argued that a 
direct transition from a metallic phase to an insulating 
and incompressible U(l) spin liquid is possible 5-8 . Yet 
the understanding of this transition remains rudimentary 
and is unconfirmed by experiment. We argue that with 
ultracold mixtures of bosons and fermions it is possible to 
study a similar transition directly. An essential clement 
of the transition to the U(l) spin liquid, the coupling to 
a deconfined (7(1) gauge field, is, however, missing. 

Model - For simplicity we confine ourselves to spinless 
fermions mixed with a single species of bosons in three di- 
mensions (d = 3) , described by the generalized Hubbard 
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FIG. 1: Schematic phase diagram of the Bose Fermi mixture 
at combined integer filling. At weak interaction the bosons 
form a BEC and the fermions form a Fermi liquid, unsta- 
ble at very low temperature to p-wave pairing. At strong 
interactions the system goes into the mixed Mott phase, in 
which composite neutral fermions (with respect to total den- 
sity) still exist as low energy degrees of freedom. Depending 
on parameters they can form either a Fermi liquid (FL) a p- 
wave superfluid or phase separate. The nature of the Mott 
critical point depends on how it is approached, by tuning the 
boson-fermion or the boson-boson interactions. 
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Here b\ and cj create bosonic and fermionic atoms respec- 
tively on site i of the lattice and um (jifi) are the bosonic 
(fermionic) site occupations. Mott phases can occur for 
commensurate filling (we take (n^) + = 1). 

When the interactions are large enough, Ubf t/,tb 
and Ubb h, fluctuations of the density are strongly 
suppressed and the system is deep in the Mott phase. 
After eliminating perturbatively the empty and doubly 
occupied states one obtains a purely fermionic model 2 
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where t cS = t f t b /U bf and V eS = (t\ + t 2 f )/U bf - 2t 2 /U bb . 
Here we have taken the singly occupied bosonic site as 
the reference vacuum | fi ) . The auxiliary fermion cj = 
fjbi is a quadratic operator in the original fields with 
c\ | fi) = fjbi | f2) . The ground state of the fermion 
model (2) can be a Fermi liquid for V e g > 0, a p-wave 
superfluid for small or moderatly large negative V e s, or 
be unstable to phase separation for large negative V e g, 
\Ve«\ 3* t e g- A phase transition out of the Mott state is 
driven by reducing the strength of the interactions U bb , 
U b f, or both. 

Consider first tuning the transition by changing U b f, 
while U bb remains very large. For the resulting hard-core 
bosons it is useful to rewrite the problem in terms of 
holes in the Mott insulator, h\ = fy. The condition of 
unity filling reads now (n^) = and the repulsive 

interaction is mapped to attraction, U b f — > —Ubf. In 
these variables, the Mott transition can be understood 
as binding of bosons to fermions, with the Mott state 
being a Fermi liquid of the molecules cj | Ct ) = /j - h\ | ) 
as V e g > in this limit. 

A similar transition, from a Fermi surface of atoms 
to a Fermi surface of molecules, has been considered in 
Ref. [9] for a Bose-Fermi mixture in the continuum with 
an interspecies Feshbach resonance. Our lattice model 
with U bb ^> U b f maps to this continuum problem for low 
densities, (n/j) <C 1. We therefore expect two transitions 
as found in [9] . First a Fermi sea of molecules starts to 
form beyond a critical value of the attraction U b f = U c \ 
and coexists with the atomic Fermi sea and with a BEC 
of the unbound bosons. The volume of the molecular 
Fermi surface grows continuously until it reaches the full 
Luttingcr volume, corresponding to the full fermion den- 
sity, at U b f = U C 2- At this point the condensate fraction 
vanishes and the system has reached the Mott insulating 
state. As pointed out in Ref. [9], interactions are irrele- 
vant at such a QCP in d = 3 and due to the quadratic 
dispersion at the bottom of the bosonic and fermionic 
bands, u) oc fc 2 , the dynamical critical exponent is z — 2. 
The same theory can be applied for nearly unity filling by 
the fermions if we apply the particle hole transformation 
on the fermions rather than the bosons. Other phases 
with broken lattice symmetry are possible at certain in- 
termediate fillings 10,11 . 

We now turn to the main focus of this paper and con- 
sider the transition driven by reducing the boson-boson 
interaction U bb while keeping U b f large. In this case we 
can eliminate the fermionic doublon state /T&||0) at 
energy ~ U b f. Using again the single boson state | f2) 
as the reference vacuum, we introduce besides the sin- 
gle fermion state c] | O ) and bosonic hole h\\ft) defined 
above also the bosonic doublon p\\n) = 2-5b||fi). 



The Hamiltonian (1) projected to low energies becomes 

H eS = ^u bb y^x n pj + n hj) - mj n a (3) 
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supplemented with the hard-core condition n p i + rihi + 
n C i < 1 on each site. U cc = (tf + t 2 )/U b f describes 
an effective nearest-neighbor repulsion of the composite 
fermions. Unity filling implies (n^i) = (n p i) and the tran- 
sition from the Bose-Fermi Mott state to the superfluid 
is described as a simultaneous condensation of the dou- 
blons and holes just as in a conventional bosonic Mott- 
superffuid transition 12 . 

Critical theory - To investigate the quantum critical 
point we analyze the corresponding critical theory. The 
most general action allowed by symmetry (including only 
the most relevant terms) is given by 

S = S b + Sf + S b f 

S b = J dxd T \d T (j)\ 2 + c 2 \Vdp\ 2 +r\dp\ 2 +u b \cj)\ A 

Sf = J ip(d T + vp ■ ( — iV — kp))ip — v,f^p\/{p ■ vp\7ip 

S b f = u b f j ' dxdr i)ip\(f)\ 2 (4) 

Here the bosonic order parameter field is rel ated to the 
particle-hole fluctuations through <j)(x) ~ y/\/c{h(x) + 
p'(x)). As in the conventional bosonic Mott transition, 
the condition (n^i) — (npi) entails the absence of a linear 
time derivatives 4>*d T (f>. Formally the same theory was 
considered by Yang 13 to address the Mott transition at 
integer boson filling in contact with a fermi sea at a non- 
commensurate filling. The crucial difference here is that 
in our theory the tj) fermions are not the physical atoms 
but rather composite objects consisting of a fermion atom 
and a bosonic hole. In the supplementary material we 
discuss how the parameters in Eq. (4) are related to the 
original microscopic model. 

The bosonic sector of the field theory (4) is identi- 
cal to that of the commensurate Mott transition in the 
purely bosonic system characterized by a linear energy- 
momentum relation, lj ~ ck, resulting in z — 1. We ana- 
lyze the coupling of the bosonic to the fermionic sector by 
performing a scaling analysis using the decoupled fixed 
point Uf = and u b f = as the reference point. Assum- 
ing tf <C tb, the couplings uj and u b f remain small. As 
the Fermi surface is two-dimensional while <fr condenses 
only at q — there is no unique way to perform such 
a scaling analysis. Using in d = 3 isotropic scaling, 
r — » Xf, t — > Ar, 4> — > 4>/\ and iji — > ijj/b 3 ^ 2 , suggests 
that Ubf is irrelevant, u b t — > u b f/X. Also alternative 
scaling schemes which take into account the curvature of 
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the Fermi surface (following, e.g.. Ref. [14]) and direct 
calculation of leading diagrams show that Ubf is irrele- 
vant. This does, however, not imply that it can be set to 
zero as Ubf can generate a marginal long-ranged interac- 
tion of the bosons, see below. Uf is marginal and leads 
in the attractive case, Uf > 0, to p-wave superfluidity. 

We first assume, that the pairing instability of the 
Fermi surface can be neglected as either uf is repulsive or 
so small that the transition temperature is smaller than 
T. Due to the irrelevance of Ub c one can integrate out 
the fermions perturbatively to obtain a purely bosonic 
theory with a modified quartic interaction term 
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The ui and q dependence of the new interaction vertex 
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where z/(0) is the fermion density of states. The local 
interaction u also receives a correction, u w uj, + u\. 

To investigate the fate of the critical point we set up a 
perturbative renormalization group (RG) by integrating 
out momenta with A/6 < \q\ < A. A rescaling, k — > k/b 
and uj — > u>/b z , restores the original cutoff A. Due to 
the u) dependence of the interactions, already to one-loop 
order one obtains self energy corrections which are ab- 
sorbed by rescaling of the field <f> and a correction to the 
dynamical critical exponent z to keep the quadratic part 
of the action fixed. A complication of the scheme is that 
higher-order long range terms of the form u n f{oj/vfq) n 
are generated during the RG flow. The scaling is there- 
fore determined by coupled equations for the dimension- 
less coupling constants g n (b) — u n (b)/(8ir 2 c 3 ) and the 
dimensionless Fermi velocity 77(6) = vp(b)/c. 
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In solving for the flow we keep terms with n < n max 
and find that the resulting flow converges with n max , 
keeping about 10 terms is enough in practice. Typical RG 
trajectories are shown in Fig. 2. While higher couplings 
g n and the r] dependence is quantitatively important, the 
qualitative structure of the flow can be understood by 
considering only go and g\ for fixed rj. Initially, both go 



FIG. 2: RG flow for three different values of the bare Fermi 
velocity vf/c using n max — 13 (see text). The inset is a 
zoom near g — showing that for the three larger val- 
ues of vf/c the local interaction is driven to negative val- 
ues before the flow is cut off at the scale of the pairing gap 
A — eF exp(— 8ttc/vf)- For these values we expect a fluctua- 
tion driven first order transition. 



and gi drop but the flow of g\ is much slower due to its 
non-local nature. Therefore gi is finite when g reaches 
zero, driving go to negative values through the last term 
in the flow equation (8)for go, thus leading to a first order 
transition. 

In the RG approach described above we have neglected 
the induced attraction Uf between fermions, which would 
lead to a pairing instability and opening of a gap A 
Ep exp(— 8ttc/v f ) in the Fermi surface. Such a gap will 
suppress the non-local coupling terms at low energies 
cA/b < A. Therefore, if the coupling constants g n have 
not yet driven the local coupling g Q to negative values 
at that scale, the first order transition will be avoided. 
Numerical solution of the flow equations suggest that 
this is the case if the bare ratio vp/c < 0.18, while for 
vp/c > 0.18 we expect a first order transition. In either 
case the Fermi surface is expected to give way to a small 
p-wave gap near the transition for T — > 0. 

Our analysis applies to a number of other quantum 
critical points where bosonic and fermionic degrees of 
freedom coexist. Consider, for example, a metallic com- 
mensurate antiferromagnet, where the shape of the Fermi 
surface is such, that the ordering wave vector Q does not 
connect different parts of the Fermi surface. In such a 
situation, there is no linear coupling, (j>tp'ip, of the order 
parameter tf> to low-energy fermions and only a quadratic 
coupling of fermions to a z = 1 bosonic QCP survives. 
Our analysis shows that in d = 3 this coupling will render 
the quantum critical point always weakly first order as 
long as no superconductivity gaps out the Fermi surface. 

Pseudogap - We now discuss the experimental ramifi- 
cations of the quantum phase transition focussing on the 
spectral function associated with emission of a fermionic 
atom in RF spectroscopy 15,16 . The crucial point to note 
is that in the long-wave length limit the physical atomic 
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FIG. 3: (a) Spectral function A(k,ui) of the fermions at 
k = kF in the superfluid phase, critical point and Mott insu- 
lator (r = —0.04,0,0.04, c/vf = 3). Inset: local density of 
states A(uj) = j dkA(k, ui) in the three cases, (b) The spectral 
function A(k,uS) at the critical point. 



fermions fi are composite objects i n te rms of the weakly 
coupled fields tp and (j>, f(x) ~ y / l/ccj}(x)ip(x) as fi — 
h\ci. Hence, the spectral function should be found from 
the Green's function Q (x, t) = (^(x, T)(f>*(x, r)c?!>(0)^)(0)}. 

For the sake of this discussion we ignore all loga- 
rithmic corrections which ultimately lead either to p- 
wave pairing or the fluctuation induced first order tran- 
sition. These subtle effects are only noticeable at ex- 
ponentially energies. The salient features of the spec- 
tral function at higher energies (or temperatures) are 
captured within the Gaussian theory obtained from ex- 
pansion about the saddle point of (4), which implies 
G(x, t) = G^,(x, tYD^{x, t), where is the free fermion 
Green's function associated with the composite fermions. 

In the superfluid side, we can take a Bogoliubov expan- 
sion of the order parameter (f> — </> + 5<pi + iSfo, to split 
the bosonic component of the Green's function into three 
contributions: T> c j ) (x,T) — \<fio\ 2 + T>i(x, r) + T>2(x, r). 
The condensate part \4>q\ 2 , when combined with the free 
fermion Green's function G.0, will give a delta function 
contribution of magnitude \4>q\ 2 dispersing with the free 
fermion dispersion. The phonon contribution leads to a 
continuous spectrum rising linearly with u up from the 
energy of the quasi-particle peak. Another continuous 
contribution onsets above the energy gap of the ampli- 
tude (or Higgs) mode. All three features are seen Fig. 
3(a), where the spectral function A(k,uj) at k = kp has 
been calculated for c/vf = 3 and a quadratic fermionic 
dispersion efc = k 2 /2. 

Upon approaching the critical point, the quasiparticle 
weight Z ~ |0o| 2 ~ U£ b — Ubb and the associated dis- 
continuity in the momentum distribution function of the 
fermions (n£) decrease to zero. Correspondingly, a pseu- 
dogap develops in the local density of states, see inset of 
Fig. 3(a). Directly at the critical point, Ubb = U§ b , where 
Z — 0, the spectral function at k — kp rises linearly in 
w, see Fig. 3(a). The underlying quadratic dispersion of 
the composite fermions and the linear dispersion of the 
bosonic excitations are clearly visible in Fig. 3(b), which 
shows A(k,Lo) at criticality, with k varying along a radial 
direction. 



Finally, inside the Mott phase the bosonic fluctuations 
can be treated as a free massive field. Hence upon con- 
volution with the fermion Green's function one obtains a 
fully gapped spectral function despite the existence of a 
gapless Fermi liquid. 

The gapless fermions that exist in the Mott insulator 
are hidden from standard single particle probes such as 
photoemission or the momentum distribution measured 
in time of flight. Interestingly however, the hidden Fermi 
surface can be revealed by noise correlations in time of 
flight images 17 . The boson-fermion cross correlations at 
momenta k and k + q are directly proportional to the 
momentum-q distribution of the composite fermions, 

K)«E< n k+q n k)-(^ +q )^)- (8) 
k 

The approximation becomes exact deep in the Mott in- 
sulating state, where fjbj — 5ijc\ for [/&/, Ubb oo. 

The composite fermions can also be seen through 
Bragg or lattice modulation probes that couple with dif- 
ferent strength to the bosons and fermions. In this case 
they effectively couple to the composite fermions. The 
appropriate structure factors will therefore display a gap- 
less spectrum in the composite Fermi liquid phase. Onset 
of p-wave pairing of the composite fermions, expected in 
certain regions of the Mott phase, will be seen as opening 
of a gap with a singular spectrum at the gap edge. 

Conclusions - Mixed boson-fermion systems in optical 
lattices open a new route, for both theoretical and exper- 
imental investigation of unconventional Mott transitions 
that entail the destruction of fermionic quasi-particles 
and the emergence of hidden Fermi surfaces of composite 
particles. Our theory accounts for the critical behavior 
of the single fermion spectral function and gives a simple 
and tractable example for the emergence of a pseudogap 
in a strongly correlated system. 

At very low energy scales, coupling of the critical 
modes to the Fermi surface can either lead to p-wave 
pairing of the composite Fermi surface or can destabilize 
the critical point leading to a fluctuation induced first or- 
der transition. While for d = 3, the nature of the phase 
transition depends on the ratio the interactions, see Fig. 
1, it has been shown that in d = 1 always a single z — 1 
transition is expected 18 . 

An interesting open question is the nature of the possi- 
ble tri-critical point postulated in Fig. 1 where Ubb ~ Ubf 
and the two z = 2 transitions meet with the z — 1 critical 
point. Interestingly, the generalized Hubbard model (1) 
exhibits supersymmetry at Ubb = Ubf, fJ-b = M/ an d 
tf = tb 19 ■ The ground state at the super-symmetric point 
has no fermions but may help to elucidate the nature of 
the tri-critical point in the limit of low fermion density. 
Another open question concerns the Mott transition in a 
commensurate mixture of bosons and spinfull fermions. 
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Supplementary material: 

I. PARAMETERS OF THE EFFECTIVE FIELD 
THEORY 

Derivation of the field theory from the lattice model (3) 
involves integration over the short wavelength modes up 
to an intermediate scale Aql, much smaller than the bare 
cutoff Ao (set by the inverse lattice spacing a) , but much 
larger than the inverse correlation length of the QCP. In 
general, it is difficult to obtain precise values of the pa- 
rameters of the effective low-energy theory, but at least 
for small fermion density, (n p i) -C I, it is simple to es- 
timate their order of magnitude, r ~ Ubb — U bb is de- 
termined by the distance to critical interaction strength 
E/X, of the order of the bosonic bandwidth, which also 
determines the velocity of the critical mode, c ~ tb- The 
hard core interaction between the particles at scale A is 



renormalized to a finite effective repulsion, Ub w 87r 2 c 3 , 
given by the scattering T-matrices between bosons at the 
scale Agl (we have taken \n(A / Aql) ~ 1). The kinetic 
energy of the fermions is generated from the correlated 
hopping term proportional to tf in (3) following integra- 
tion over the high energy boson holes and is proportional 
to the hole density in the lattice which is a finite num- 
ber of order I at the QCP. In other words the effective 
mass of the Fermions is 1/m* ~ tf( n h) ~ tf. Using 
the Free fermion propagator thus generated, we can now 
find the magnitude of the boson-fermion vertex in the 
same way as we did for the bose-bose vertex and obtain 
Ubf ~ 8nc/m*. Finally, the integration over the holes 
also generates an effective p-wave attraction between the 
fermions, Uf ~ l/(87rm^c) for tt <C t b . The dimension- 
less interaction strength is then Ufv(Ep) ~ VF/{Snc) for 
(/ « ((, and of order 1 for tf > tt- For Ubf ^> Ubb this at- 
tractive interaction will be larger than the small repulsive 
interaction arising from U cc in Eq. (3). 



